An unstabilized video camera often produces a video image that bounces and jerks according to the movement of the camera user during filming. In order to remedy this normally undesirable result, digital video stabilization is used as a low-cost alternative to optical stabilization, removing video “jitter” through image post-processing, thus making the video appear as if the camera had been stabilized during use. Although many video stabilization techniques have been proposed and are currently used with charge-coupled device (CCD) imaging devices, very few techniques have been proposed for use with complementary metal oxide semiconductor (CMOS) imaging devices, such as those widely used in digital cameras and camera phones.
Many CMOS imaging devices acquire digital images using a rolling shutter mechanism. In a rolling shutter device, a pixel array is incrementally exposed to light from an image, row-by-row. For example, the top or first row in the pixel array may be exposed first to light from an image. Soon thereafter, the second row is exposed, followed by the third row, and so forth. Each pixel row has a same total exposure time, but each row begins its exposure time (and hence also ends its exposure time) at different times. Most significantly, there is a significant delay between the moment the first row is exposed to light and the moment the last or bottom row is exposed to light. The result of this delay is a rolling shutter artifact that appears in an acquired image when either the camera or the viewed object moves during the course of the rolling exposure, as illustrated in FIGS. 1A-1E.
FIG. 1A depicts an image 5 as it would appear if the object depicted in the image 5 and the camera taking the image 5 were both not moving. The image 5 appears to be very much like the actual object captured by the image 5. However, if either the camera pans to the left or the object moves to the right during a rolling shutter exposure, the image 5 appears distorted, as seen in FIG. 1B. The rolling shutter results in different portions of the object being acquired for the image 5 at different times. The movement of the object during the exposure time results in some portions of the object appearing to move ahead of other portions of the object. In the example depicted in FIG. 1B, the camera panning to the left or the object moving to the right results in the bottom portion of the object appearing to move further to the right than the top portion of the object in the image 5, because the bottom portion is exposed at a later time than the top portion. Similarly, in FIG. 1C, when the camera pans to the right or the object moves to the left, the bottom portion of the object appears to move further to the left than the top portion of the object. In FIG. 1D, when the camera tilts upward or the object moves downward, the object in the image 5 appears to stretch. When the camera tilts downward or the object moves upward, the object appears to shrink in the image 5, as depicted in FIG. 1E.
The above-described rolling shutter artifacts are especially noticeable when conventional video stabilization techniques are applied to captured video. Even after the video has been stabilized, meaning that the up and down, side-to-side or rotational movements of an unstabilized camera have been compensated for, the distortion caused by the rolling shutter is still apparent. Camera movements from side to side result in the filmed object appearing to wobble instead of standing still, even after video stabilization. Up and down movements of the camera result in the continuous stretching and shrinking of the filmed object. Thus, video stabilization for a rolling shutter video requires both removal of the rolling shutter artifacts as well as the application of video stabilization techniques.
One technique for stabilizing rolling shutter video uses a 3×3 transformation matrix to model the image relationship between a rolling shutter image and a non-rolling shutter image. The transformation matrix is used to transform the acquired rolling shutter image to an equivalent non-rolling shutter image. Afterwards, camera motion is smoothed. Unfortunately, the 3×3 transformation matrix/model makes two critical assumptions that often are not correct. First, the model assumes that image differences between two consecutive frames are the result of a simple translation in x and y coordinates. The model ignores rotational movement. Additionally, the model assumes that any translational motion is uniform from the time that the top row is captured to the time the bottom row is captured. However, motion tends to change abruptly from frame to frame as well as during frame acquisition. Thus, although the assumptions used in the translation model are indeed rough approximations of actual camera or object movement, the approximations are not always valid, and thus do not fully compensate for the rolling shutter artifacts in video.